Counter-Factual Reinforcement Learning: How to Model Decision-Makers That Anticipate the Future

This chapter introduces a novel framework for modeling interacting humans in a multi-stage game. This “iterated semi network-form game” framework has the following desirable characteristics: (1) Bounded rational players, (2) strategic players (i.e., players account for one another’s reward functions when predicting one another’s behavior), and (3) computational tractability even on real-world systems. We achieve these benefits by combining concepts from game theory and reinforcement learning. To be precise, we extend the bounded rational “level-K reasoning” model to apply to games over multiple stages. Our extension allows the decomposition of the overall modeling problem into a series of smaller ones, each Ritchie Lee Carnegie Mellon University Silicon Valley, NASA Ames Research Park, Mail Stop 23-11, Moffett Field, CA, 94035 e-mail: [email protected] David H. Wolpert Santa Fe Institute, 1399 Hyde Park Rd., Santa Fe, NM 87501 Los Alamos National Laboratory, MS B256, Los Alamos, NM, 87545 e-mail: [email protected] James Bono American University, 4400 Massachusetts Ave. NW, Washington DC 20016 e-mail: [email protected] Scott Backhaus Los Alamos National Laboratory, MS K764, Los Alamos, NM 87545 e-mail: [email protected]